{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "W2_Homework.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/CIS-522/course-content/blob/main/tutorials/W02_PyTorchDLN/student/W2_Homework.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hwKKo5_FzsXP"
      },
      "source": [
        "# CIS-522 Week 2 Homework\n",
        "\n",
        "\n",
        "**Instructor:** Konrad Kording\n",
        "\n",
        "**Content Creators:** Ameet Rahane, Spiros Chavlis"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q2t00IxoJbXI"
      },
      "source": [
        "---\n",
        "## Preface\n",
        "Since this week's homework requires coding, we recommend saving this notebook in your google Drive (`File -> Save a copy in Drive`), and share the link to the final version in the subscription airtable form. You can also attach the code to the form if you prefer off-colab coding."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8kdBhPZj3mPh",
        "cellView": "form"
      },
      "source": [
        "#@markdown What is your Pennkey and pod? (text, not numbers, e.g. bfranklin)\n",
        "my_pennkey = '' #@param {type:\"string\"}\n",
        "my_pod = 'Select' #@param ['Select', 'euclidean-wombat', 'sublime-newt', 'buoyant-unicorn', 'lackadaisical-manatee','indelible-stingray','superfluous-lyrebird','discreet-reindeer','quizzical-goldfish','ubiquitous-cheetah','nonchalant-crocodile','fashionable-lemur','spiffy-eagle','electric-emu','quotidian-lion','astute-jellyfish', 'quantum-herring']\n",
        "\n",
        "# start timing\n",
        "import time\n",
        "try:t0;\n",
        "except NameError: t0 = time.time()\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N7s6B1jx0djA"
      },
      "source": [
        "## Part 1. Recap\n",
        "This week, we looked at how to create deep fully connected networks in pytorch, covering everything from an introduction to Pytorch tensor operations to deep linear networks. Meet with your pod for 10 minutes to discuss what you learned, what was clear, and what you hope to learn more about.\n",
        "\n",
        "In this homework, we will be exploring that a little more. First, we will read a paper by Andrew M. Saxe, James L. McClelland, and Surya Ganguli on _A mathematical theory of semantic development in deep neural networks_. Then, you will design your own linear network to solve a given problem on a dataset.\n",
        "\n",
        "**Note:** Before or after doing the Homework, we highly encourage you to finish the optional material of [W2_Tutorial2](https://colab.research.google.com/github/CIS-522/course-content/blob/main/tutorials/W2_PyTorchDLN/student/W2_Tutorial2.ipynb) notebook, and especially Section 7. Although the concepts covered in the optional part are not a requirement to finish the Homework, finishing these materials will help you improve your current intuition on Deep Learning.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oDjCjSsS02dD"
      },
      "source": [
        "## Part 2: Read a paper\n",
        "\n",
        "Read the paper [A mathematical theory of semantic development in deep neural networks](https://www.pnas.org/content/116/23/11537.short). Take some time and try to contextualize this with respect to the content we covered this week. For more context, watch the talk by Surya Ganguli below. After reading the paper, discuss the main figures with your pod."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-z_Jom06QqSv",
        "cellView": "form"
      },
      "source": [
        "#@markdown ### Dr. Ganguli's talk\r\n",
        "from IPython.display import YouTubeVideo\r\n",
        "video = YouTubeVideo(id=\"SJAvFqHw-hA\", width=854, height=480, fs=1)\r\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\r\n",
        "\r\n",
        "video"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OjpbnkQUGr1B"
      },
      "source": [
        "#@markdown **Student response**: What do you think is the most important aspect of the paper?\n",
        "thoughts_on_paper = '' #@param {type:\"string\"}\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cFb-xRYWV5YC"
      },
      "source": [
        "## Part 3: Design your deep network!\r\n",
        "\r\n",
        "In this section, you'll design a deep linear network to perform linear regression in a real dataset. We will provide the data for you.\r\n",
        "\r\n",
        "We will use the **QSAR fish toxicity dataset** taken from the UCI Machine Learning Repository for this homework. For more information you can see [here](https://archive.ics.uci.edu/ml/datasets/QSAR+fish+toxicity) and also read the relevant [publication](http://www.doi.org/10.1080/1062936X.2015.1018938). This study focuses on the development of a new QSAR (Quantitative structure-activity relationship, for more info see [here](https://en.wikipedia.org/wiki/Quantitative_structure%E2%80%93activity_relationship)) model to predict the acute toxicity of various chemicals, defined as $LC_{50}$ 96 hours, towards the fathead minnow (Pimephales promelas).\r\n",
        "\r\n",
        "The dataset consists of $908$ number of examples, $6$ features, and $1$ target variable.\r\n",
        "\r\n",
        "Namely, the features are six molecular descriptors:\r\n",
        "\r\n",
        "1. CIC0: Complementary Information Content index, neighborhood symmetry of 0-order (encode information regarding heteroatoms)\r\n",
        "2. SM1_Dz(Z): the spectral moment of order one from Barysz matrix weighted by atomic number (account for heteroatoms, large values more heterogeneity)\r\n",
        "3. GATS1i: Geary autocorrelation of lag 1 weighted by ionization potential (shows the number of carbon-carbon bonds)\r\n",
        "4. NdsCH: Number of atoms of type dsCH (encode information\r\n",
        "about the electrophilic characteristics of chemicals)\r\n",
        "5. NdssC: Number of atoms of type dssC (encode information\r\n",
        "about the electrophilic characteristics of chemicals)\r\n",
        "6. MLogP: Moriguchi octanol-water partition coefficient (a widely accepted estimate of the lipophilicity of organic compounds, which is considered the driving force of narcosis) \r\n",
        "\r\n",
        "and the target variable is a quantitative response (toxicity), $LC_{50} [-log_{10}(mol/L)]$.\r\n",
        "\r\n",
        "This small project aims to find the linear relationship between these molecular markers and the toxicity. So, you will implement a deep linear network to perform linear regression.\r\n",
        "\r\n",
        "Towards this goal, you have to perform the following steps:\r\n",
        "\r\n",
        "1. Download and visualize the dataset. As our samples contain six features, we can perform PCA or any other dimensionality reduction method to visualize our dataset, e.g., [sklearn.decomposition.PCA](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html)\r\n",
        "\r\n",
        "  ```python\r\n",
        "    # Code example for PCA\r\n",
        "    X = np.array([[-1, -1],\r\n",
        "                  [-2, -1],\r\n",
        "                  [-3, -2],\r\n",
        "                  [1, 1],\r\n",
        "                  [2, 1],\r\n",
        "                  [3, 2]])\r\n",
        "\r\n",
        "    pca = decomposition.PCA(n_components=2)  # project in 2D space\r\n",
        "    pca.fit(X)\r\n",
        "    Xprojected = pca.transform(X)\r\n",
        "  ```\r\n",
        "\r\n",
        "2. Split the dataset into training and testing subsets (you can split in 80-20 scheme) using [sklearn.model_selection.train_test_split](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html)\r\n",
        "3. Write your deep linear network with any approach that is more comfortable for you. Here, we provide an example using the `nn.Module` class of PyTorch.\r\n",
        "\r\n",
        "  ```python\r\n",
        "    # Example\r\n",
        "    class Network(nn.Module):\r\n",
        "\r\n",
        "      def __init__(self, input_dim, hidden_1, hidden_2, output_dim):\r\n",
        "        super(Network, self).__init__()\r\n",
        "\r\n",
        "        self.input_dim = input_dim\r\n",
        "        self.hidden_1 = hidden_1\r\n",
        "        self.output_dim = output_dim\r\n",
        "\r\n",
        "        # A fully-connected network (FCN) with 2 hidden layers\r\n",
        "        self.fc1 = nn.Linear(self.input_dim, self.hidden_1)\r\n",
        "        self.fc2 = nn.Linear(self.hidden_1, self.output_dim)\r\n",
        "\r\n",
        "      def forward(self, x):\r\n",
        "        h1 = self.fc1(x)\r\n",
        "        out = self.fc3(h1)\r\n",
        "\r\n",
        "        return out\r\n",
        "\r\n",
        "  ```\r\n",
        "\r\n",
        "4. Choose the hyperparameters (i.e., number of epochs, learning rate, number of layers and nodes per layer)\r\n",
        "\r\n",
        "  ```python\r\n",
        "    # Here, you have to write your training loop\r\n",
        "\r\n",
        "    def training(inputs, targets, num_epochs, learning_rate):\r\n",
        "\r\n",
        "      # Set up the network\r\n",
        "      mynetwork = Network(...)\r\n",
        "\r\n",
        "      # optimizer and loss function\r\n",
        "      criterion = ...\r\n",
        "      optimizer = ...\r\n",
        "\r\n",
        "      for epoch in range(num_epochs):\r\n",
        "        # Your CODE goes here\r\n",
        "\r\n",
        "      return (loss_training, loss_test, ...)\r\n",
        "    \r\n",
        "  ```\r\n",
        "\r\n",
        "5. Train the model and give the graph with\r\n",
        "  - training and test loss vs. number of epochs\r\n",
        "  ```python\r\n",
        "    plt.figure()\r\n",
        "    plt.plot(..., label='training', 'b')\r\n",
        "    plt.plot(..., label='test', 'r')\r\n",
        "    plt.legend()\r\n",
        "    plt.show()\r\n",
        "  ```\r\n",
        "6. Give a plot with the model's performance (actual vs. predicted targets)\r\n",
        "  ```python\r\n",
        "    plt.figure()\r\n",
        "    plt.plot(..., alpha=0.5)\r\n",
        "    plt.plot(..., label='regression', 'r')  # plot the y=y_hat line\r\n",
        "    plt.legend()\r\n",
        "    plt.show()  \r\n",
        "\r\n",
        "  ```\r\n",
        "7. Train the model again by removing each time one feature. Report what do you observe in terms of performance. You can use NumPy's function [`np.delete()`](https://numpy.org/doc/stable/reference/generated/numpy.delete.html) to remove specific rows/columns from a `ndarray` object.\r\n",
        "  ```python\r\n",
        "    # Here, we have to remove one by one the features from the X dataset\r\n",
        "    # Hint: The network input layer has to be changed ...\r\n",
        "    # retrain and keep the predictions.\r\n",
        "    # Then, plot all regressions in one plot using subplots\r\n",
        "    plt.figure()\r\n",
        "    for i, preds in enumerate(predictions):\r\n",
        "      plt.subplot(2, 3, i + 1)\r\n",
        "      plt.plot(...)  # plot original y vs. y_hat\r\n",
        "      plt.plot(...)  # y=y_hat\r\n",
        "      plt.legend()\r\n",
        "  ```\r\n",
        "\r\n",
        "\r\n",
        "*Hints:*\r\n",
        "- Be sure to use the correct dimensions\r\n",
        "- Put legends and title in all plots\r\n",
        "\r\n",
        "**Note:** Typically, to make unbiased and more accurate predictions, we perform multiple training pieces with different initializations (recall that we initialize parameters sampling from distribution) and different train-test splits. Here we want to run one time for the shake of timing, but feel free to run more times the training loops and plot the average losses!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xfWUk2O_MepY",
        "cellView": "form",
        "outputId": "fbb05bb6-5782-4932-9aa3-bedc42573693",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "#@markdown #Run this cell to download the dataset\n",
        "#@markdown ##Inputs are stored in `X`, and targets in `y`.\n",
        "\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "\n",
        "url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/00504/qsar_fish_toxicity.csv'\n",
        "\n",
        "headers = ['CIC0', 'SM1_Dz(Z)', 'GATS1i', 'NdsCH', 'NdssC', 'MLogP', 'LC50']\n",
        "\n",
        "df1 = pd.read_csv(url, names = headers, header=None, delimiter=\";\")\n",
        "\n",
        "X = np.array(df1)[:,:-1]\n",
        "y = np.array(df1)[:,-1].reshape(-1,1)\n",
        "\n",
        "print(f'Data: \\n{df1}')\n",
        "\n",
        "print(f'\\ninput shape of X: {X.shape}, ' \n",
        "      f'targets shape of y: {y.shape}')"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Data: \n",
            "      CIC0  SM1_Dz(Z)  GATS1i  NdsCH  NdssC  MLogP   LC50\n",
            "0    3.260      0.829   1.676      0      1  1.453  3.770\n",
            "1    2.189      0.580   0.863      0      0  1.348  3.115\n",
            "2    2.125      0.638   0.831      0      0  1.348  3.531\n",
            "3    3.027      0.331   1.472      1      0  1.807  3.510\n",
            "4    2.094      0.827   0.860      0      0  1.886  5.390\n",
            "..     ...        ...     ...    ...    ...    ...    ...\n",
            "903  2.801      0.728   2.226      0      2  0.736  3.109\n",
            "904  3.652      0.872   0.867      2      3  3.983  4.040\n",
            "905  3.763      0.916   0.878      0      6  2.918  4.818\n",
            "906  2.831      1.393   1.077      0      1  0.906  5.317\n",
            "907  4.057      1.032   1.183      1      3  4.754  8.201\n",
            "\n",
            "[908 rows x 7 columns]\n",
            "\n",
            "input shape of X: (908, 6), targets shape of y: (908, 1)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ktven-uHHed1"
      },
      "source": [
        "Insert your code in the cell below or use any IDE (e.g, [PyCharm](https://www.jetbrains.com/pycharm/), [VS-Code](https://code.visualstudio.com/), [Spyder](https://www.spyder-ide.org/), etc.) or editor you prefer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tw05SVeOHbWd"
      },
      "source": [
        "# Code Cell\n",
        "\n",
        "### YOUR CODE GOES HERE"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "khy9nAjzKl1h"
      },
      "source": [
        "#@markdown **Student response**: Which is the most/less important feature of this dataset?\n",
        "dataset_features = '' #@param {type:\"string\"}\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kPyknvoP2y6T"
      },
      "source": [
        "# Submission\n",
        "Double check that the answers have been correctly formatted in the Airtable below, and edit them as desired. **In addition to submitting the form, contribute and reflect on these readings in the student Slack space.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X1avaLZ7Ltcs"
      },
      "source": [
        "Feel free to use the embeded form below or use this link:\n",
        "<a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://airtable.com/shrV231jtyUrwkvuZ\">https://airtable.com/shrV231jtyUrwkvuZ</a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "anCjE03r3cJ6",
        "cellView": "form"
      },
      "source": [
        "#@markdown #Run Cell to Show Airtable Form\n",
        "#@markdown ##**Confirm your answers and then click \"Submit\"**\n",
        "import numpy as np\n",
        "import time\n",
        "from IPython.display import IFrame\n",
        "def prefill_form(src, fields: dict):\n",
        "  '''\n",
        "  src: the original src url to embed the form\n",
        "  fields: a dictionary of field:value pairs,\n",
        "  e.g. {\"pennkey\": my_pennkey, \"location\": my_location}\n",
        "  '''\n",
        "  prefills = \"&\".join([\"prefill_%s=%s\"%(key, fields[key]) for key in fields])\n",
        "  src = src + prefills\n",
        "  src = \"+\".join(src.split(\" \"))\n",
        "  return src\n",
        "\n",
        "#autofill fields if they are not present\n",
        "#a missing pennkey and pod will result in an Airtable warning\n",
        "#which is easily fixed user-side.\n",
        "try: my_pennkey;\n",
        "except NameError: my_pennkey = \"\"\n",
        "try: my_pod;\n",
        "except NameError: my_pod = \"Select\"\n",
        "try: general_thoughts;\n",
        "except NameError: general_thoughts = \"\"\n",
        "try: dataset_features;\n",
        "except NameError: dataset_features = \"\"\n",
        "\n",
        "times = 0\n",
        "\n",
        "fields = {\"pennkey\": my_pennkey,\n",
        "          \"pod\": my_pod,\n",
        "          \"thoughts_on_paper\":general_thoughts,\n",
        "          \"reflection_1\":dataset_features,\n",
        "          \"cumulative_times\": times}\n",
        "\n",
        "src = \"https://airtable.com/embed/shrV231jtyUrwkvuZ?\"\n",
        "\n",
        "#now instead of the original source url, we do: src = prefill_form(src, fields)\n",
        "display(IFrame(src = prefill_form(src, fields), width = 800, height = 400))"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}